System and method for augmented and virtual reality

ABSTRACT

One embodiment is directed to a system for enabling two or more users to interact within a virtual world comprising virtual world data, comprising a computer network comprising one or more computing devices, the one or more computing devices comprising memory, processing circuitry, and software stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data; wherein at least a first portion of the virtual world data originates from a first user virtual world local to a first user, and wherein the computer network is operable to transmit the first portion to a user device for presentation to a second user, such that the second user may experience the first portion from the location of the second user, such that aspects of the first user virtual world are effectively passed to the second user.

RELATED APPLICATION DATA

This is a continuation application of U.S. patent application Ser. No.17/083,255 filed on Oct. 28, 2020, which is a continuation of U.S.patent application Ser. No. 16/752,577, filed on Jan. 24, 2020, now U.S.Pat. No. 10,862,930, which is a continuation of U.S. application Ser.No. 16/659,415 filed on Oct. 21, 2019, now U.S. Pat. No. 10,587,659,which is a continuation of U.S. patent application Ser. No. 16/261,352filed on Jan. 29, 2019, now U.S. Pat. No. 10,469,546, which is acontinuation of U.S. patent application Ser. No. 15/920,201, filed onMar. 13, 2018 now Abandoned, which is a continuation of U.S. patentapplication Ser. No. 15/238,657, filed on Aug. 16, 2016 now U.S. Pat.No. 10,021,149, which is a continuation application of U.S. patentapplication Ser. No. 14/965,169 filed Dec. 10, 2015 now abandoned, whichis a continuation of U.S. patent application Ser. No. 14/514,115 filedOct. 14, 2014 now abandoned, which is a continuation application of U.S.patent application Ser. No. 13/663,466 filed Oct. 29, 2012 now U.S. Pat.No. 9,215,293, which claims the benefit under 35 U.S.C. § 119 to U.S.Provisional Applications Ser. No. 61/552,941 filed Oct. 28, 2011. Theforegoing applications are hereby incorporated by reference into thepresent application in their entirety.

FIELD OF THE INVENTION

The present invention generally relates to systems and methodsconfigured to facilitate interactive virtual or augmented realityenvironments for one or more users.

BACKGROUND

Virtual and augmented reality environments are generated by computersusing, in part, data that describes the environment. This data maydescribe, for example, various objects with which a user may sense andinteract with. Examples of these objects include objects that arerendered and displayed for a user to see, audio that is played for auser to hear, and tactile (or haptic) feedback for a user to feel. Usersmay sense and interact with the virtual and augmented realityenvironments through a variety of visual, auditory and tactical means.

SUMMARY

One embodiment is directed to a system for enabling two or more users tointeract within a virtual world comprising virtual world data,comprising a computer network comprising one or more computing devices,the one or more computing devices comprising memory, processingcircuitry, and software stored at least in part in the memory andexecutable by the processing circuitry to process at least a portion ofthe virtual world data; wherein at least a first portion of the virtualworld data originates from a first user virtual world local to a firstuser, and wherein the computer network is operable to transmit the firstportion to a user device for presentation to a second user, such thatthe second user may experience the first portion from the location ofthe second user, such that aspects of the first user virtual world areeffectively passed to the second user. The first and second users may bein different physical locations or in substantially the same physicallocation. At least a portion of the virtual world may be configured tochange in response to a change in the virtual world data. At least aportion of the virtual world may be configured to change in response toa physical object sensed by the user device. The change in virtual worlddata may represent a virtual object having a predetermined relationshipwith the physical object. The change in virtual world data may bepresented to a second user device for presentation to the second useraccording to the predetermined relationship. The virtual world may beoperable to be rendered by at least one of the computer servers or auser device. The virtual world may be presented in a two-dimensionalformat. The virtual world may be presented in a three-dimensionalformat. The user device may be operable to provide an interface forenabling interaction between a user and the virtual world in anaugmented reality mode. The user device may be operable to provide aninterface for enabling interaction between a user and the virtual worldin a virtual reality mode. The user device may be operable to provide aninterface for enabling interaction between a user and the virtual worlda combination of augmented and virtual reality mode. The virtual worlddata may be transmitted over a data network. The computer network may beoperable to receive at least a portion of the virtual world data from auser device. At least a portion of the virtual world data transmitted tothe user device may comprise instructions for generating at least aportion of the virtual world. At least a portion of the virtual worlddata may be transmitted to a gateway for at least one of processing ordistribution. At least one of the one or more computer servers may beoperable to process virtual world data distributed by the gateway.

Another embodiment is directed to a system for virtual and/or augmenteduser experience wherein remote avatars are animated based at least inpart upon data on a wearable device with optional input from voiceinflection and facial recognition software.

Another embodiment is directed to a system for virtual and/or augmenteduser experience wherein a camera pose or viewpoint position and vectormay be placed anywhere in a world sector.

Another embodiment is directed to a system for virtual and/or augmenteduser experience wherein worlds or portions thereof may be rendered forobserving users at diverse and selectable scales.

Another embodiment is directed to a system for virtual and/or augmenteduser experience wherein features, such as points or parametric lines, inaddition to pose tagged images, may be utilized as base data for a worldmodel from which software robots, or object recognizers, may be utilizedto create parametric representations of real-world objects, taggingsource features for mutual inclusion in segmented objects and the worldmodel.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a representative embodiment of the disclosed systemfor facilitating interactive virtual or augmented reality environmentsfor multiple users.

FIG. 2 illustrates an example of a user device for interacting with thesystem illustrated in FIG. 1.

FIG. 3 illustrates an example embodiment of a mobile, wearable userdevice.

FIG. 4 illustrates an example of objects viewed by a user when themobile, wearable user device of FIG. 3 is operating in an augmentedmode.

FIG. 5 illustrates an example of objects viewed by a user when themobile, wearable user device of FIG. 3 is operating in a virtual mode.

FIG. 6 illustrates an example of objects viewed by a user when themobile, wearable user device of FIG. 3 is operating in a blended virtualinterface mode.

FIG. 7 illustrates an embodiment wherein two users located in differentgeographical locations each interact with the other user and a commonvirtual world through their respective user devices.

FIG. 8 illustrates an embodiment wherein the embodiment of FIG. 7 isexpanded to include the use of a haptic device.

FIG. 9A illustrates an example of mixed mode interfacing, wherein afirst user is interfacing a digital world in a blended virtual interfacemode and a second user is interfacing the same digital world in avirtual reality mode.

FIG. 9B illustrates another example of mixed mode interfacing, whereinthe first user is interfacing a digital world in a blended virtualinterface mode and the second user is interfacing the same digital worldin an augmented reality mode.

FIG. 10 illustrates an example illustration of a user's view wheninterfacing the system in an augmented reality mode.

FIG. 11 illustrates an example illustration of a user's view showing avirtual object triggered by a physical object when the user isinterfacing the system in an augmented reality mode.

FIG. 12 illustrates one embodiment of an augmented and virtual realityintegration configuration wherein one user in an augmented realityexperience visualizes the presence of another user in a virtual realtyexperience.

FIG. 13 illustrates one embodiment of a time and/or contingency eventbased augmented reality experience configuration.

FIG. 14 illustrates one embodiment of a user display configurationsuitable for virtual and/or augmented reality experiences.

FIG. 15 illustrates one embodiment of local and cloud-based computingcoordination.

FIG. 16 illustrates various aspects of registration configurations.

DETAILED DESCRIPTION

Referring to FIG. 1, system 100 is representative hardware forimplementing processes described below. This representative systemcomprises a computing network 105 comprised of one or more computerservers 110 connected through one or more high bandwidth interfaces 115.The servers in the computing network need not be co-located. The one ormore servers 110 each comprise one or more processors for executingprogram instructions. The servers also include memory for storing theprogram instructions and data that is used and/or generated by processesbeing carried out by the servers under direction of the programinstructions.

The computing network 105 communicates data between the servers 110 andbetween the servers and one or more user devices 120 over one or moredata network connections 130. Examples of such data networks include,without limitation, any and all types of public and private datanetworks, both mobile and wired, including for example theinterconnection of many of such networks commonly referred to as theInternet. No particular media, topology or protocol is intended to beimplied by the figure.

User devices are configured for communicating directly with computingnetwork 105, or any of the servers 110. Alternatively, user devices 120communicate with the remote servers 110, and, optionally, with otheruser devices locally, through a specially programmed, local gateway 140for processing data and/or for communicating data between the network105 and one or more local user devices 120.

As illustrated, gateway 140 is implemented as a separate hardwarecomponent, which includes a processor for executing softwareinstructions and memory for storing software instructions and data. Thegateway has its own wired and/or wireless connection to data networksfor communicating with the servers 110 comprising computing network 105.Alternatively, gateway 140 can be integrated with a user device 120,which is worn or carried by a user. For example, the gateway 140 may beimplemented as a downloadable software application installed and runningon a processor included in the user device 120. The gateway 140provides, in one embodiment, one or more users access to the computingnetwork 105 via the data network 130.

Servers 110 each include, for example, working memory and storage forstoring data and software programs, microprocessors for executingprogram instructions, graphics processors and other special processorsfor rendering and generating graphics, images, video, audio andmulti-media files. Computing network 105 may also comprise devices forstoring data that is accessed, used or created by the servers 110.

Software programs running on the servers and optionally user devices 120and gateways 140, are used to generate digital worlds (also referred toherein as virtual worlds) with which users interact with user devices120. A digital world is represented by data and processes that describeand/or define virtual, non-existent entities, environments, andconditions that can be presented to a user through a user device 120 forusers to experience and interact with. For example, some type of object,entity or item that will appear to be physically present wheninstantiated in a scene being viewed or experienced by a user mayinclude a description of its appearance, its behavior, how a user ispermitted to interact with it, and other characteristics. Data used tocreate an environment of a virtual world (including virtual objects) mayinclude, for example, atmospheric data, terrain data, weather data,temperature data, location data, and other data used to define and/ordescribe a virtual environment. Additionally, data defining variousconditions that govern the operation of a virtual world may include, forexample, laws of physics, time, spatial relationships and other datathat may be used to define and/or create various conditions that governthe operation of a virtual world (including virtual objects).

The entity, object, condition, characteristic, behavior or other featureof a digital world will be generically referred to herein, unless thecontext indicates otherwise, as an object (e.g., digital object, virtualobject, rendered physical object, etc.). Objects may be any type ofanimate or inanimate object, including but not limited to, buildings,plants, vehicles, people, animals, creatures, machines, data, video,text, pictures, and other users. Objects may also be defined in adigital world for storing information about items, behaviors, orconditions actually present in the physical world. The data thatdescribes or defines the entity, object or item, or that stores itscurrent state, is generally referred to herein as object data. This datais processed by the servers 110 or, depending on the implementation, bya gateway 140 or user device 120, to instantiate an instance of theobject and render the object in an appropriate manner for the user toexperience through a user device.

Programmers who develop and/or curate a digital world create or defineobjects, and the conditions under which they are instantiated. However,a digital world can allow for others to create or modify objects. Oncean object is instantiated, the state of the object may be permitted tobe altered, controlled or manipulated by one or more users experiencinga digital world.

For example, in one embodiment, development, production, andadministration of a digital world is generally provided by one or moresystem administrative programmers. In some embodiments, this may includedevelopment, design, and/or execution of story lines, themes, and eventsin the digital worlds as well as distribution of narratives throughvarious forms of events and media such as, for example, film, digital,network, mobile, augmented reality, and live entertainment. The systemadministrative programmers may also handle technical administration,moderation, and curation of the digital worlds and user communitiesassociated therewith, as well as other tasks typically performed bynetwork administrative personnel.

Users interact with one or more digital worlds using some type of alocal computing device, which is generally designated as a user device120. Examples of such user devices include, but are not limited to, asmart phone, tablet device, heads-up display (HUD), gaming console, orany other device capable of communicating data and providing aninterface or display to the user, as well as combinations of suchdevices. In some embodiments, the user device 120 may include, orcommunicate with, local peripheral or input/output components such as,for example, a keyboard, mouse, joystick, gaming controller, hapticinterface device, motion capture controller, an optical tracking devicesuch as those available from Leap Motion, Inc., or those available fromMicrosoft under the tradename Kinect®, audio equipment, voice equipment,projector system, 3D display, and holographic 3D contact lens.

An example of a user device 120 for interacting with the system 100 isillustrated in FIG. 2. In the example embodiment shown in FIG. 2, a user210 may interface one or more digital worlds through a smart phone 220.The gateway is implemented by a software application 230 stored on andrunning on the smart phone 220. In this particular example, the datanetwork 130 includes a wireless mobile network connecting the userdevice (i.e., smart phone 220) to the computer network 105.

In one implementation of preferred embodiment, system 100 is capable ofsupporting a large number of simultaneous users (e.g., millions ofusers), each interfacing with the same digital world, or with multipledigital worlds, using some type of user device 120.

The user device provides to the user an interface for enabling a visual,audible, and/or physical interaction between the user and a digitalworld generated by the servers 110, including other users and objects(real or virtual) presented to the user. The interface provides the userwith a rendered scene that can be viewed, heard or otherwise sensed, andthe ability to interact with the scene in real-time. The manner in whichthe user interacts with the rendered scene may be dictated by thecapabilities of the user device. For example, if the user device is asmart phone, the user interaction may be implemented by a usercontacting a touch screen. In another example, if the user device is acomputer or gaming console, the user interaction may be implementedusing a keyboard or gaming controller. User devices may includeadditional components that enable user interaction such as sensors,wherein the objects and information (including gestures) detected by thesensors may be provided as input representing user interaction with thevirtual world using the user device.

The rendered scene can be presented in various formats such as, forexample, two-dimensional or three-dimensional visual displays (includingprojections), sound, and haptic or tactile feedback. The rendered scenemay be interfaced by the user in one or more modes including, forexample, augmented reality, virtual reality, and combinations thereof.The format of the rendered scene, as well as the interface modes, may bedictated by one or more of the following: user device, data processingcapability, user device connectivity, network capacity and systemworkload. Having a large number of users simultaneously interacting withthe digital worlds, and the real-time nature of the data exchange, isenabled by the computing network 105, servers 110, the gateway component140 (optionally), and the user device 120.

In one example, the computing network 105 IS comprised of a large-scalecomputing system having single and/or multi-core servers (i.e., servers110) connected through high-speed connections (e.g., high bandwidthinterfaces 115). The computing network 105 may form a cloud or gridnetwork. Each of the servers includes memory, or is coupled withcomputer readable memory for storing software for implementing data tocreate, design, alter, or process objects of a digital world. Theseobjects and their instantiations may be dynamic, come in and out ofexistence, change over time, and change in response to other conditions.Examples of dynamic capabilities of the objects are generally discussedherein with respect to various embodiments. In some embodiments, eachuser interfacing the system 100 may also be represented as an object,and/or a collection of objects, within one or more digital worlds.

The servers 110 within the computing network 105 also storecomputational state data for each of the digital worlds. Thecomputational state data (also referred to herein as state data) may bea component of the object data, and generally defines the state of aninstance of an object at a given instance in time. Thus, thecomputational state data may change over time and may be impacted by theactions of one or more users and/or programmers maintaining the system100. As a user impacts the computational state data (or other datacomprising the digital worlds), the user directly alters or otherwisemanipulates the digital world. If the digital world is shared with, orinterfaced by, other users, the actions of the user may affect what isexperienced by other users interacting with the digital world. Thus, insome embodiments, changes to the digital world made by a user will beexperienced by other users interfacing with the system 100.

The data stored in one or more servers 110 within the computing network105 is, in one embodiment, transmitted or deployed at a high-speed, andwith low latency, to one or more user devices 120 and/or gatewaycomponents 140. In one embodiment, object data shared by servers may becomplete or may be compressed, and contain instructions for recreatingthe full object data on the user side, rendered and visualized by theuser's local computing device (e.g., gateway 140 and/or user device120). Software running on the servers 110 of the computing network 105may, in some embodiments, adapt the data it generates and sends to aparticular user's device 120 for objects within the digital world (orany other data exchanged by the computing network 105) as a function ofthe user's specific device and bandwidth. For example, when a userinteracts with a digital world through a user device 120, a server 110may recognize the specific type of device being used by the user, thedevice's connectivity and/or available bandwidth between the user deviceand server, and appropriately size and balance the data being deliveredto the device to optimize the user interaction. An example of this mayinclude reducing the size of the transmitted data to a low resolutionquality, so that the data may be displayed on a particular user devicehaving a low resolution display. In a preferred embodiment, thecomputing network 105 and/or gateway component 140 deliver data to theuser device 120 at a rate sufficient to present an interface operatingat 15 frames/second or higher, and at a resolution that is highdefinition quality or greater.

The gateway 140 provides local connection to the computing network 105for one or more users. In some embodiments, it may be implemented by adownloadable software application that runs on the user device 120 oranother local device, such as that shown in FIG. 2. In otherembodiments, it may be implemented by a hardware component (withappropriate software/firmware stored on the component, the componenthaving a processor) that is either in communication with, but notincorporated with or attracted to, the user device 120, or incorporatedwith the user device 120. The gateway 140 communicates with thecomputing network 105 via the data network 130, and provides dataexchange between the computing network 105 and one or more local userdevices 120. As discussed in greater detail below, the gateway component140 may include software, firmware, memory, and processing circuitry,and may be capable of processing data communicated between the network105 and one or more local user devices 120.

In some embodiments, the gateway component 140 monitors and regulatesthe rate of the data exchanged between the user device 120 and thecomputer network 105 to allow optimum data processing capabilities forthe particular user device 120. For example, in some embodiments, thegateway 140 buffers and downloads both static and dynamic aspects of adigital world, even those that are beyond the field of view presented tothe user through an interface connected with the user device. In such anembodiment, instances of static objects (structured data, softwareimplemented methods, or both) may be stored in memory (local to thegateway component 140, the user device 120, or both) and are referencedagainst the local user's current position, as indicated by data providedby the computing network 105 and/or the user's device 120. Instances ofdynamic objects, which may include, for example, intelligent softwareagents and objects controlled by other users and/or the local user, arestored in a high-speed memory buffer. Dynamic objects representing atwo-dimensional or three-dimensional object within the scene presentedto a user can be, for example, broken down into component shapes, suchas a static shape that is moving but is not changing, and a dynamicshape that is changing. The part of the dynamic object that is changingcan be updated by a real-time, threaded high priority data stream from aserver 110, through computing network 105, managed by the gatewaycomponent 140. As one example of a prioritized threaded data stream,data that is within a 60 degree field-of-view of the user's eye may begiven higher priority than data that is more peripheral. Another exampleincludes prioritizing dynamic characters and/or objects within theuser's field-of-view over static objects in the background.

In addition to managing a data connection between the computing network105 and a user device 120, the gateway component 140 may store and/orprocess data that may be presented to the user device 120. For example,the gateway component 140 may, in some embodiments, receive compresseddata describing, for example, graphical objects to be rendered forviewing by a user, from the computing network 105 and perform advancedrendering techniques to alleviate the data load transmitted to the userdevice 120 from the computing network 105. In another example, in whichgateway 140 is a separate device, the gateway 140 may store and/orprocess data for a local instance of an object rather than transmittingthe data to the computing network 105 for processing.

Referring now also to FIG. 3, the digital worlds may be experienced byone or more users in various formats that may depend upon thecapabilities of the user's device. In some embodiments, the user device120 may include, for example, a smart phone, tablet device, heads-updisplay (HUD), gaming console, or a wearable device. Generally, the userdevice will include a processor for executing program code stored inmemory on the device, coupled with a display, and a communicationsinterface. An example embodiment of a user device is illustrated in FIG.3, wherein the user device comprises a mobile, wearable device, namely ahead-mounted display system 300. In accordance with an embodiment of thepresent disclosure, the head-mounted display system 300 includes a userinterface 302, user-sensing system 304, environment-sensing system 306,and a processor 308. Although the processor 308 is shown in FIG. 3 as anisolated component separate from the head-mounted system 300, in analternate embodiment, the processor 308 may be integrated with one ormore components of the head-mounted system 300, or may be integratedinto other system 100 components such as, for example, the gateway 140.

The user device presents to the user an interface 302 for interactingwith and experiencing a digital world. Such interaction may involve theuser and the digital world, one or more other users interfacing thesystem 100, and objects within the digital world. The interface 302generally provides image and/or audio sensory input (and in someembodiments, physical sensory input) to the user. Thus, the interface302 may include speakers (not shown) and a display component 303capable, in some embodiments, of enabling stereoscopic 3D viewing and/or3D viewing which embodies more natural characteristics of the humanvision system. In some embodiments, the display component 303 maycomprise a transparent interface (such as a clear OLED) which, when inan “off” setting, enables an optically correct view of the physicalenvironment around the user with little-to-no optical distortion orcomputing overlay. As discussed in greater detail below, the interface302 may include additional settings that allow for a variety ofvisual/interface performance and functionality.

The user-sensing system 304 may include, in some embodiments, one ormore sensors 310 operable to detect certain features, characteristics,or information related to the individual user wearing the system 300.For example, in some embodiments, the sensors 310 may include a cameraor optical detection/scanning circuitry capable of detecting real-timeoptical characteristics/measurements of the user such as, for example,one or more of the following: pupil constriction/dilation, angularmeasurement/positioning of each pupil, spherocity, eye shape (as eyeshape changes over time) and other anatomic data. This data may provide,or be used to calculate, information (e.g., the user's visual focalpoint) that may be used by the head-mounted system 300 and/or interfacesystem 100 to optimize the user's viewing experience. For example, inone embodiment, the sensors 310 may each measure a rate of pupilcontraction for each of the user's eyes. This data may be transmitted tothe processor 308 (or the gateway component 140 or to a server 110),wherein the data is used to determine, for example, the user's reactionto a brightness setting of the interface display 303. The interface 302may be adjusted in accordance with the user's reaction by, for example,dimming the display 303 if the user's reaction indicates that thebrightness level of the display 303 is too high. The user-sensing system304 may include other components other than those discussed above orillustrated in FIG. 3. For example, in some embodiments, theuser-sensing system 304 may include a microphone for receiving voiceinput from the user. The user sensing system may also include one ormore infrared camera sensors, one or more visible spectrum camerasensors, structured light emitters and/or sensors, infrared lightemitters, coherent light emitters and/or sensors, gyros, accelerometers,magnetometers, proximity sensors, GPS sensors, ultrasonic emitters anddetectors and haptic interfaces.

The environment-sensing system 306 includes one or more sensors 312 forobtaining data from the physical environment around a user. Objects orinformation detected by the sensors may be provided as input to the userdevice. In some embodiments, this input may represent user interactionwith the virtual world. For example, a user viewing a virtual keyboardon a desk may gesture with his fingers as if he were typing on thevirtual keyboard. The motion of the fingers moving may be captured bythe sensors 312 and provided to the user device or system as input,wherein the input may be used to change the virtual world or create newvirtual objects. For example, the motion of the fingers may berecognized (using a software program) as typing, and the recognizedgesture of typing may be combined with the known location of the virtualkeys on the virtual keyboard. The system may then render a virtualmonitor displayed to the user (or other users interfacing the system)wherein the virtual monitor displays the text being typed by the user.

The sensors 312 may include, for example, a generally outward-facingcamera or a scanner for interpreting scene information, for example,through continuously and/or intermittently projected infrared structuredlight. The environment-sensing system 306 may be used for mapping one ormore elements of the physical environment around the user by detectingand registering the local environment, including static objects, dynamicobjects, people, gestures and various lighting, atmospheric and acousticconditions. Thus, in some embodiments, the environment-sensing system306 may include image-based 3D reconstruction software embedded in alocal computing system (e.g., gateway component 140 or processor 308)and operable to digitally reconstruct one or more objects or informationdetected by the sensors 312. In one exemplary embodiment, theenvironment-sensing system 306 provides one or more of the following:motion capture data (including gesture recognition), depth sensing,facial recognition, object recognition, unique object featurerecognition, voice/audio recognition and processing, acoustic sourcelocalization, noise reduction, infrared or similar laser projection, aswell as monochrome and/or color CMOS sensors (or other similar sensors),field-of-view sensors, and a variety of other optical-enhancing sensors.It should be appreciated that the environment-sensing system 306 mayinclude other components other than those discussed above or illustratedin FIG. 3. For example, in some embodiments, the environment-sensingsystem 306 may include a microphone for receiving audio from the localenvironment. The user sensing system may also include one or moreinfrared camera sensors, one or more visible spectrum camera sensors,structure light emitters and/or sensors, infrared light emitters,coherent light emitters and/or sensors gyros, accelerometers,magnetometers, proximity sensors, GPS sensors, ultrasonic emitters anddetectors and haptic interfaces.

As mentioned above, the processor 308 may, in some embodiments, beintegrated with other components of the head-mounted system 300,integrated with other components of the interface system 100, or may bean isolated device (wearable or separate from the user) as shown in FIG.3. The processor 308 may be connected to various components of thehead-mounted system 300 and/or components of the interface system 100through a physical, wired connection, or through a wireless connectionsuch as, for example, mobile network connections (including cellulartelephone and data networks), Wi-Fi or Bluetooth. The processor 308 mayinclude a memory module, integrated and/or additional graphicsprocessing unit, wireless and/or wired internet connectivity, and codecand/or firmware capable of transforming data from a source (e.g., thecomputing network 105, the user-sensing system 304, theenvironment-sensing system 306, or the gateway component 140) into imageand audio data, wherein the images/video and audio may be presented tothe user via the interface 302.

The processor 308 handles data processing for the various components ofthe headmounted system 300 as well as data exchange between thehead-mounted system 300 and the gateway component 140 and, in someembodiments, the computing network 105. For example, the processor 308may be used to buffer and process data streaming between the user andthe computing network 105, thereby enabling a smooth, continuous andhigh fidelity user experience. In some embodiments, the processor 308may process data at a rate sufficient to achieve anywhere between 8frames/second at 320×240 resolution to 24 frames/second at highdefinition resolution (1280×720), or greater, such as 60-120frames/second and 4 k resolution and higher (10 k+resolution and 50,000frames/second). Additionally, the processor 308 may store and/or processdata that may be presented to the user, rather than streamed inreal-time from the computing network 105. For example, the processor 308may, in some embodiments, receive compressed data from the computingnetwork 105 and perform advanced rendering techniques (such as lightingor shading) to alleviate the data load transmitted to the user device120 from the computing network 105. In another example, the processor308 may store and/or process local object data rather than transmittingthe data to the gateway component 140 or to the computing network 105.

The head-mounted system 300 may, in some embodiments, include varioussettings, or modes, that allow for a variety of visual/interfaceperformance and functionality. The modes may be selected manually by theuser, or automatically by components of the head-mounted system 300 orthe gateway component 140. As previously mentioned, one example ofheadmounted system 300 includes an “off” mode, wherein the interface 302provides substantially no digital or virtual content. In the off mode,the display component 303 may be transparent, thereby enabling anoptically correct view of the physical environment around the user withlittle-to-no optical distortion or computing overlay.

In one example embodiment, the head-mounted system 300 includes an“augmented” mode, wherein the interface 302 provides an augmentedreality interface. In the augmented mode, the interface display 303 maybe substantially transparent, thereby allowing the user to view thelocal, physical environment. At the same time, virtual object dataprovided by the computing network 105, the processor 308, and/or thegateway component 140 is presented on the display 303 in combinationwith the physical, local environment.

FIG. 4 illustrates an example embodiment of objects viewed by a userwhen the interface 302 is operating in an augmented mode. As shown inFIG. 4, the interface 302 presents a physical object 402 and a virtualobject 404. In the embodiment illustrated in FIG. 4, the physical object402 is a real, physical object existing in the local environment of theuser, whereas the virtual object 404 is an object created by the system100, and displayed via the user interface 302. In some embodiments, thevirtual object 404 may be displayed at a fixed position or locationwithin the physical environment (e.g., a virtual monkey standing next toa particular street sign located in the physical environment), or may bedisplayed to the user as an object located at a position relative to theuser interface/display 303 (e.g., a virtual clock or thermometer visiblein the upper, left corner of the display 303).

In some embodiments, virtual objects may be made to be cued off of, ortrigged by, an object physically present within or outside a user'sfield of view. Virtual object 404 is cued off, or triggered by, thephysical object 402. For example, the physical object 402 may actuallybe a stool, and the virtual object 404 may be displayed to the user(and, in some embodiments, to other users interfacing the system 100) asa virtual animal standing on the stool. In such an embodiment, theenvironment-sensing system 306 may use software and/or firmware stored,for example, in the processor 308 to recognize various features and/orshape patterns (captured by the sensors 312) to identify the physicalobject 402 as a stool. These recognized shape patterns such as, forexample, the stool top, may be used to trigger the placement of thevirtual object 404. Other examples include walls, tables, furniture,cars, buildings, people, floors, plants, animals—any object which can beseen can be used to trigger an augmented reality experience in somerelationship to the object or objects.

In some embodiments, the particular virtual object 404 that is triggeredmay be selected by the user or automatically selected by othercomponents of the head-mounted system 300 or interface system 100.Additionally, in embodiments in which the virtual object 404 isautomatically triggered, the particular virtual object 404 may beselected based upon the particular physical object 402 (or featurethereof) off which the virtual object 404 is cued or triggered. Forexample, if the physical object is identified as a diving boardextending over a pool, the triggered virtual object may be a creaturewearing a snorkel, bathing suit, floatation device, or other relateditems.

In another example embodiment, the head-mounted system 300 may include a“virtual” mode, wherein the interface 302 provides a virtual realityinterface. In the virtual mode, the physical environment is omitted fromthe display 303, and virtual object data provided by the computingnetwork 105, the processor 308, and/or the gateway component 140 ispresented on the display 303. The omission of the physical environmentmay be accomplished by physically blocking the visual display 303 (e.g.,via a cover) or through a feature of the interface 302 wherein thedisplay 303 transitions to an opaque setting. In the virtual mode, liveand/or stored visual and audio sensory may be presented to the userthrough the interface 302, and the user experiences and interacts with adigital world (digital objects, other users, etc.) through the virtualmode of the interface 302. Thus, the interface provided to the user inthe virtual mode is comprised of virtual object data comprising avirtual, digital world.

FIG. 5 illustrates an example embodiment of a user interface when theheadmounted interface 302 is operating in a virtual mode. As shown inFIG. 5, the user interface presents a virtual world 500 comprised ofdigital objects 510, wherein the digital objects 510 may includeatmosphere, weather, terrain, buildings, and people. Although it is notillustrated in FIG. 5, digital objects may also include, for example,plants, vehicles, animals, creatures, machines, artificial intelligence,location information, and any other object or information defining thevirtual world 500.

In another example embodiment, the head-mounted system 300 may include a“blended” mode, wherein various features of the head-mounted system 300(as well as features of the virtual and augmented modes) may be combinedto create one or more custom interface modes. In one example custominterface mode, the physical environment is omitted from the display303, and virtual object data is presented on the display 303 in a mannersimilar to the virtual mode. However, in this example custom interfacemode, virtual objects may be fully virtual (i.e., they do not exist inthe local, physical environment) or they may be real, local, physicalobjects rendered as a virtual object in the interface 302 in place ofthe physical object. Thus, in this particular custom mode (referred toherein as a blended virtual interface mode), live and/or stored visualand audio sensory may be presented to the user through the interface302, and the user experiences and interacts with a digital worldcomprising fully virtual objects and rendered physical objects.

FIG. 6 illustrates an example embodiment of a user interface operatingin accordance with the blended virtual interface mode. As shown in FIG.6, the user interface presents a virtual world 600 comprised of fullyvirtual objects 610, and rendered physical objects 620 (renderings ofobjects otherwise physically present in the scene). In accordance withthe example illustrated in FIG. 6, the rendered physical objects 620include a building 620A, ground 620B, and a platform 620C, and are shownwith a bolded outline 630 to indicate to the user that the objects arerendered. Additionally, the fully virtual objects 610 include anadditional user 610A, clouds 610B, sun 610C, and flames 610D on top ofthe platform 620C. It should be appreciated that fully virtual objects610 may include, for example, atmosphere, weather, terrain, buildings,people, plants, vehicles, animals, creatures, machines, artificialintelligence, location information, and any other object or informationdefining the virtual world 600, and not rendered from objects existingin the local, physical environment. Conversely, the rendered physicalobjects 620 are real, local, physical objects rendered as a virtualobject in the interface 302. The bolded outline 630 represents oneexample for indicating rendered physical objects to a user. As such, therendered physical objects may be indicated as such using methods otherthan those disclosed herein.

In some embodiments, the rendered physical objects 620 may be detectedusing the sensors 312 of the environment-sensing system 306 (or usingother devices such as a motion or image capture system), and convertedinto digital object data by software and/or firmware stored, forexample, in the processing circuitry 308. Thus, as the user interfaceswith the system 100 in the blended virtual interface mode, variousphysical objects may be displayed to the user as rendered physicalobjects. This may be especially useful for allowing the user tointerface with the system 100, while still being able to safely navigatethe local, physical environment. In some embodiments, the user may beable to selectively remove or add the rendered physical objects to theinterface display 303.

In another example custom interface mode, the interface display 303 maybe substantially transparent, thereby allowing the user to view thelocal, physical environment, while various local, physical objects aredisplayed to the user as rendered physical objects. This example custominterface mode is similar to the augmented mode, except that one or moreof the virtual objects may be rendered physical objects as discussedabove with respect to the previous example.

The foregoing example custom interface modes represent a few exampleembodiments of various custom interface modes capable of being providedby the blended mode of the head-mounted system 300. Accordingly, variousother custom interface modes may be created from the various combinationof features and functionality provided by the components of theheadmounted system 300 and the various modes discussed above withoutdeparting from the scope of the present disclosure.

The embodiments discussed herein merely describe a few examples forproviding an interface operating in an off, augmented, virtual, orblended mode, and are not intended to limit the scope or content of therespective interface modes or the functionality of the components of thehead-mounted system 300. For example, in some embodiments, the virtualobjects may include data displayed to the user (time, temperature,elevation, etc.), objects created and/or selected by the system 100,objects created and/or selected by a user, or even objects representingother users interfacing the system 100. Additionally, the virtualobjects may include an extension of physical objects (e.g., a virtualsculpture growing from a physical platform) and may be visuallyconnected to, or disconnected from, a physical object.

The virtual objects may also be dynamic and change with time, change inaccordance with various relationships (e.g., location, distance, etc.)between the user or other users, physical objects, and other virtualobjects, and/or change in accordance with other variables specified inthe software and/or firmware of the head-mounted system 300, gatewaycomponent 140, or servers 110. For example, in certain embodiments, avirtual object may respond to a user device or component thereof (e.g.,a virtual ball moves when a haptic device is placed next to it),physical or verbal user interaction (e.g., a virtual creature runs awaywhen the user approaches it, or speaks when the user speaks to it), achair is thrown at a virtual creature and the creature dodges the chair,other virtual objects (e.g., a first virtual creature reacts when itsees a second virtual creature), physical variables such as location,distance, temperature, time, etc. or other physical objects in theuser's environment (e.g., a virtual creature shown standing in aphysical street becomes flattened when a physical car passes).

The various modes discussed herein may be applied to user devices otherthan the head-mounted system 300. For example, an augmented realityinterface may be provided via a mobile phone or tablet device. In suchan embodiment, the phone or tablet may use a camera to capture thephysical environment around the user, and virtual objects may beoverlaid on the phone/tablet display screen. Additionally, the virtualmode may be provided by displaying the digital world on the displayscreen of the phone/tablet. Accordingly, these modes may be blended asto create various custom interface modes as described above using thecomponents of the phone/tablet discussed herein, as well as othercomponents connected to, or used in combination with, the user device.For example, the blended virtual interface mode may be provided by acomputer monitor, television screen, or other device lacking a cameraoperating in combination with a motion or image capture system. In thisexample embodiment, the virtual world may be viewed from themonitor/screen and the object detection and rendering may be performedby the motion or image capture system.

FIG. 7 illustrates an example embodiment of the present disclosure,wherein two users located in different geographical locations eachinteract with the other user and a common virtual world through theirrespective user devices. In this embodiment, the two users 701 and 702are throwing a virtual ball 703 (a type of virtual object) back andforth, wherein each user is capable of observing the impact of the otheruser on the virtual world (e.g., each user observes the virtual ballchanging directions, being caught by the other user, etc.). Since themovement and location of the virtual objects (i.e., the virtual ball703) are tracked by the servers 110 in the computing network 105, thesystem 100 may, in some embodiments, communicate to the users 701 and702 the exact location and timing of the arrival of the ball 703 withrespect to each user. For example, if the first user 701 is located inLondon, the user 701 may throw the ball 703 to the second user 702located in Los Angeles at a velocity calculated by the system 100.Accordingly, the system 100 may communicate to the second user 702(e.g., via email, text message, instant message, etc.) the exact timeand location of the ball's arrival. As such, the second user 702 may usehis device to see the ball 703 arrive at the specified time and located.One or more users may also use geo-location mapping software (orsimilar) to track one or more virtual objects as they travel virtuallyacross the globe. An example of this may be a user wearing a 3Dhead-mounted display looking up in the sky and seeing a virtual planeflying overhead, superimposed on the real world. The virtual plane maybe flown by the user, by intelligent software agents (software runningon the user device or gateway), other users who may be local and/orremote, and/or any of these combinations.

As previously mentioned, the user device may include a haptic interfacedevice, wherein the haptic interface device provides a feedback (e.g.,resistance, vibration, lights, sound, etc.) to the user when the hapticdevice is determined by the system 100 to be located at a physical,spatial location relative to a virtual object. For example, theembodiment described above with respect to FIG. 7 may be expanded toinclude the use of a haptic device 802, as shown in FIG. 8.

In this example embodiment, the haptic device 802 may be displayed inthe virtual world as a baseball bat. When the ball 703 arrives, the user702 may swing the haptic device 802 at the virtual ball 703. If thesystem 100 determines that the virtual bat provided by the haptic device802 made “contact” with the ball 703, then the haptic device 802 mayvibrate or provide other feedback to the user 702, and the virtual ball703 may ricochet off the virtual bat in a direction calculated by thesystem 100 in accordance with the detected speed, direction, and timingof the ball-to-bat contact.

The disclosed system 100 may, in some embodiments, facilitate mixed modeinterfacing, wherein multiple users may interface a common virtual world(and virtual objects contained therein) using different interface modes(e.g., augmented, virtual, blended, etc.). For example, a first userinterfacing a particular virtual world in a virtual interface mode mayinteract with a second user interfacing the same virtual world in anaugmented reality mode.

FIG. 9A illustrates an example wherein a first user 901 (interfacing adigital world of the system 100 in a blended virtual interface mode) andfirst object 902 appear as virtual objects to a second user 922interfacing the same digital world of the system 100 in a full virtualreality mode. As described above, when interfacing the digital world viathe blended virtual interface mode, local, physical objects (e.g., firstuser 901 and first object 902) may be scanned and rendered as virtualobjects in the virtual world. The first user 901 may be scanned, forexample, by a motion capture system or similar device, and rendered inthe virtual world (by software/firmware stored in the motion capturesystem, the gateway component 140, the user device 120, system servers110, or other devices) as a first rendered physical object 931.Similarly, the first object 902 may be scanned, for example, by theenvironment-sensing system 306 of a head-mounted interface 300, andrendered in the virtual world (by software/firmware stored in theprocessor 308, the gateway component 140, system servers 110, or otherdevices) as a second rendered physical object 932. The first user 901and first object 902 are shown in a first portion 910 of FIG. 9A asphysical objects in the physical world. In a second portion 920 of FIG.9A, the first user 901 and first object 902 are shown as they appear tothe second user 922 interfacing the same digital world of the system 100in a full virtual reality mode: as the first rendered physical object931 and second rendered physical object 932.

FIG. 9B illustrates another example embodiment of mixed modeinterfacing, wherein the first user 901 is interfacing the digital worldin a blended virtual interface mode, as discussed above, and the seconduser 922 is interfacing the same digital world (and the second user'sphysical, local environment 925) in an augmented reality mode. In theembodiment in FIG. 9B, the first user 901 and first object 902 arelocated at a first physical location 915, and the second user 922 islocated at a different, second physical location 925 separated by somedistance from the first location 915. In this embodiment, the virtualobjects 931 and 932 may be transposed in realtime (or near real-time) toa location within the virtual world corresponding to the second location925. Thus, the second user 922 may observe and interact, in the seconduser's physical, local environment 925, with the rendered physicalobjects 931 and 932 representing the first user 901 and first object902, respectively.

FIG. 10 illustrates an example illustration of a user's view wheninterfacing the system 100 in an augmented reality mode. As shown inFIG. 10, the user sees the local, physical environment (i.e., a cityhaving multiple buildings) as well as a virtual character 1010 (i.e.,virtual object). The position of the virtual character 1010 may betriggered by a 2D visual target (for example, a billboard, postcard ormagazine) and/or one or more 3D reference frames such as buildings,cars, people, animals, airplanes, portions of a building, and/or any 3Dphysical object, virtual object, and/or combinations thereof. In theexample illustrated in FIG. 10, the known position of the buildings inthe city may provide the registration fiducials and/or information andkey features for rendering the virtual character 1010. Additionally, theuser's geospatial location (e.g., provided by GPS, attitude/positionsensors, etc.) or mobile location relative to the buildings, maycomprise data used by the computing network 105 to trigger thetransmission of data used to display the virtual character(s) 1010. Insome embodiments, the data used to display the virtual character 1010may comprise the rendered character 1010 and/or instructions (to becarried out by the gateway component 140 and/or user device 120) forrendering the virtual character 1010 or portions thereof. In someembodiments, if the geospatial location of the user is unavailable orunknown, a server 110, gateway component 140, and/or user device 120 maystill display the virtual object 1010 using an estimation algorithm thatestimates where particular virtual objects and/or physical objects maybe located, using the user's last known position as a function of timeand/or other parameters. This may also be used to determine the positionof any virtual objects should the user's sensors become occluded and/orexperience other malfunctions.

In some embodiments, virtual characters or virtual objects may comprisea virtual statue, wherein the rendering of the virtual statue istriggered by a physical object. For example, referring now to FIG. 11, avirtual statue 1110 may be triggered by a real, physical platform 1120.The triggering of the statue 1110 may be in response to a visual objector feature (e.g., fiducials, design features, geometry, patterns,physical location, altitude, etc.) detected by the user device or othercomponents of the system 100. When the user views the platform 1120without the user device, the user sees the platform 1120 with no statue1110. However, when the user views the platform 1120 through the userdevice, the user sees the statue 1110 on the platform 1120 as shown inFIG. 11. The statue 1110 is a virtual object and, therefore, may bestationary, animated, change over time or with respect to the user'sviewing position, or even change depending upon which particular user isviewing the statue 1110. For example, if the user is a small child, thestatue may be a dog; yet, if the viewer is an adult male, the statue maybe a large robot as shown in FIG. 11. These are examples of userdependent and/or state dependent experiences. This will enable one ormore users to perceive one or more virtual objects alone and/or incombination with physical objects and experience customized andpersonalized versions of the virtual objects. The statue 1110 (orportions thereof) may be rendered by various components of the systemincluding, for example, software/firmware installed on the user device.Using data indicating the location and attitude of the user device, incombination with the registration features of the virtual object (i.e.,statue 1110), the virtual object (i.e., statue 1110) forms arelationship with the physical object (i.e., platform 1120). Forexample, the relationship between one or more virtual objects with oneor more physical objects may be a function of distance, positioning,time, geo-location, proximity to one or more other virtual objects,and/or any other functional relationship that includes virtual and/orphysical data of any kind. In some embodiments, image recognitionsoftware in the user device may further enhance the digital-to-physicalobject relationship.

The interactive interface provided by the disclosed system and methodmay be implemented to facilitate various activities such as, forexample, interacting with one or more virtual environments and objects,interacting with other users, as well as experiencing various forms ofmedia content, including advertisements, music concerts, and movies.Accordingly, the disclosed system facilitates user interaction such thatthe user not only views or listens to the media content, but rather,actively participates in and experiences the media content. In someembodiments, the user participation may include altering existingcontent or creating new content to be rendered in one or more virtualworlds. In some embodiments, the media content, and/or users creatingthe content, may be themed around a mythopoeia of one or more virtualworlds.

In one example, musicians (or other users) may create musical content tobe rendered to users interacting with a particular virtual world. Themusical content may include, for example, various singles, EPs, albums,videos, short films, and concert performances. In one example, a largenumber of users may interface the system 100 to simultaneouslyexperience a virtual concert performed by the musicians.

In some embodiments, the media produced may contain a unique identifiercode associated with a particular entity (e.g., a band, artist, user,etc.). The code may be in the form of a set of alphanumeric characters,UPC codes, QR codes, 2D image triggers, 3D physical object featuretriggers, or other digital mark, as well as a sound, image, and/or both.In some embodiments, the code may also be embedded with digital mediawhich may be interfaced using the system 100. A user may obtain the code(e.g., via payment of a fee) and redeem the code to access the mediacontent produced by the entity associated with the identifier code. Themedia content may be added or removed from the user's interface.

In one embodiment, to avoid the computation and bandwidth limitations ofpassing realtime or near realtime video data from one computing systemto another with low latency, such as from a cloud computing system to alocal processor coupled to a user, parametric information regardingvarious shapes and geometries may be transferred and utilized to definesurfaces, while textures maybe transferred and added to these surfacesto bring about static or dynamic detail, such as bitmap-based videodetail of a person's face mapped upon a parametrically reproduced facegeometry. As another example, if a system is configured to recognize aperson's face, and knows that the person's avatar is located in anaugmented world, the system may be configured to pass the pertinentworld information and the person's avatar information in one relativelylarge setup transfer, after which remaining transfers to a localcomputing system, such as that 308 depicted in FIG. 1, for localrendering may be limited to parameter and texture updates, such as tomotion parameters of the person's skeletal structure and moving bitmapsof the person's face—all at orders of magnitude less bandwidth relativeto the initial setup transfer or passing of realtime video. Cloud-basedand local computing assets thus may be used in an integrated fashion,with the cloud handling computation that does not require relatively lowlatency, and the local processing assets handling tasks wherein lowlatency is at a premium, and in such case, the form of data transferredto the local systems preferably is passed at relatively low bandwidthdue to the form an amount of such data (i.e., parametric info, textures,etc. versus realtime video of everything).

Referring ahead to FIG. 15, a schematic illustrates coordination betweencloud computing assets (46) and local processing assets (308, 120). Inone embodiment, the cloud (46) assets are operatively coupled, such asvia wired or wireless networking (wireless being preferred for mobility,wired being preferred for certain high-bandwidth or high-data-volumetransfers that may be desired), directly to (40, 42) one or both of thelocal computing assets (120, 308), such as processor and memoryconfigurations which may be housed in a structure configured to becoupled to a user's head (120) or belt (308). These computing assetslocal to the user may be operatively coupled to each other as well, viawired and/or wireless connectivity configurations (44). In oneembodiment, to maintain a low-inertia and small-size head mountedsubsystem (120), primary transfer between the user and the cloud (46)may be via the link between the belt-based subsystem (308) and thecloud, with the head mounted subsystem (120) primarily data-tethered tothe belt-based subsystem (308) using wireless connectivity, such asultra-wideband (“UWB”) connectivity, as is currently employed, forexample, in personal computing peripheral connectivity applications.

With efficient local and remote processing coordination, and anappropriate display device for a user, such as the user interface 302 oruser “display device” featured in FIG. 3, the display device 14described below in reference to FIG. 14, or variations thereof, aspectsof one world pertinent to a user's current actual or virtual locationmay be transferred or “passed” to the user and updated in an efficientfashion.

Indeed, in one embodiment, with one person utilizing a virtual realitysystem (“VRS”) in an augmented reality mode and another person utilizinga VRS in a completely virtual mode to explore the same world local tothe first person, the two users may experience one another in that worldin various fashions. For example, referring to FIG. 12, a scenariosimilar to that described in reference to FIG. 11 is depicted, with theaddition of a visualization of an avatar 2 of a second user who isflying through the depicted augmented reality world from a completelyvirtual reality scenario. In other words, the scene depicted in FIG. 12may be experienced and displayed in augmented reality for the firstperson—with two augmented reality elements (the statue 1110 and theflying bumble bee avatar 2 of the second person) displayed in additionto actual physical elements around the local world in the scene, such asthe ground, the buildings in the background, the statue platform 1120.Dynamic updating may be utilized to allow the first person to visualizeprogress of the second person's avatar 2 as the avatar 2 flies throughthe world local to the first person.

Again, with a configuration as described above, wherein there is oneworld model that can reside on cloud computing resources and bedistributed from there, such world can be “passable” to one or moreusers in a relatively low bandwidth form preferable to trying to passaround realtime video data or the like. The augmented experience of theperson standing near the statue (i.e., as shown in FIG. 12) may beinformed by the cloud-based world model, a subset of which may be passeddown to them and their local display device to complete the view. Aperson sitting at a remote display device, which may be as simple as apersonal computer sitting on a desk, can efficiently download that samesection of information from the cloud and have it rendered on theirdisplay. Indeed, one person actually present in the park near the statuemay take a remotely-located friend for a walk in that park, with thefriend joining through virtual and augmented reality. The system willneed to know where the street is, wherein the trees are, where thestatue is—but with that information on the cloud, the joining friend candownload from the cloud aspects of the scenario, and then start walkingalong as an augmented reality local relative to the person who isactually in the park.

Referring to FIG. 13, a time and/or other contingency parameter basedembodiment is depicted, wherein a person is engaged with a virtualand/or augmented reality interface, such as the user interface 302 oruser display device featured in FIG. 3, the display device 14 describedbelow in reference to FIG. 14, or variations thereof, is utilizing thesystem (4) and enters a coffee establishment to order a cup of coffee(6). The VRS may be configured to utilize sensing and data gatheringcapabilities, locally and/or remotely, to provide display enhancementsin augmented and/or virtual reality for the person, such as highlightedlocations of doors in the coffee establishment or bubble windows of thepertinent coffee menu (8). When the person receives the cup of coffeethat he has ordered, or upon detection by the system of some otherpertinent parameter, the system may be configured to display (10) one ormore time-based augmented or virtual reality images, video, and/or soundin the local environment with the display device, such as a Madagascarjungle scene from the walls and ceilings, with or without jungle soundsand other effects, either static or dynamic. Such presentation to theuser may be discontinued based upon a timing parameter (i.e., 5 minutesafter the full coffee cup has been recognized and handed to the user; 10minutes after the system has recognized the user walking through thefront door of the establishment, etc) or other parameter, such as arecognition by the system that the user has finished the coffee bynoting the upside down orientation of the coffee cup as the user ingeststhe last sip of coffee from the cup—or recognition by the system thatthe user has left the front door of the establishment (12).

Referring to FIG. 14, one embodiment of a suitable user display device(14) is shown, comprising a display lens (82) which may be mounted to auser's head or eyes by a housing or frame (84). The display lens (82)may comprise one or more transparent mirrors positioned by the housing(84) in front of the user's eyes (20) and configured to bounce projectedlight (38) into the eyes (20) and facilitate beam shaping, while alsoallowing for transmission of at least some light from the localenvironment in an augmented reality configuration (in a virtual realityconfiguration, it may be desirable for the display system 14 to becapable of blocking substantially all light from the local environment,such as by a darkened visor, blocking curtain, all black LCD panel mode,or the like). In the depicted embodiment, two wide-field-of-view machinevision cameras (16) are coupled to the housing (84) to image theenvironment around the user; in one embodiment these cameras (16) aredual capture visible light/infrared light cameras. The depictedembodiment also comprises a pair of scanned-laser shaped-wavefront(i.e., for depth) light projector modules with display mirrors andoptics configured to project light (38) into the eyes (20) as shown. Thedepicted embodiment also comprises two miniature infrared cameras (24)paired with infrared light sources (26, such as light emitting diodes“LED”s), which are configured to be able to track the eyes (20) of theuser to support rendering and user input. The system (14) furtherfeatures a sensor assembly (39), which may comprise X, Y, and Z axisaccelerometer capability as well as a magnetic compass and X, Y, and Zaxis gyro capability, preferably providing data at a relatively highfrequency, such as 200 Hz. The depicted system (14) also comprises ahead pose processor (36), such as an ASIC (application specificintegrated circuit), FPGA (field programmable gate array), and/or ARMprocessor (advanced reduced-instruction-set machine), which may beconfigured to calculate real or near-real time user head pose from widefield of view image information output from the capture devices (16).Also shown is another processor (32) configured to execute digitaland/or analog processing to derive pose from the gyro, compass, and/oraccelerometer data from the sensor assembly (39). The depictedembodiment also features a GPS (37, global positioning satellite)subsystem to assist with pose and positioning. Finally, the depictedembodiment comprises a rendering engine (34) which may feature hardwarerunning a software program configured to provide rendering informationlocal to the user to facilitate operation of the scanners and imaginginto the eyes of the user, for the user's view of the world. Therendering engine (34) is operatively coupled (81, 70, 76/78, 80; i.e.,via wired or wireless connectivity) to the sensor pose processor (32),the image pose processor (36), the eye tracking cameras (24), and theprojecting subsystem (18) such that light of rendered augmented and/orvirtual reality objects is projected using a scanned laser arrangement(18) in a manner similar to a retinal scanning display. The wavefront ofthe projected light beam (38) may be bent or focused to coincide with adesired focal distance of the augmented and/or virtual reality object.The mini infrared cameras (24) may be utilized to track the eyes tosupport rendering and user input (i.e., where the user is looking, whatdepth he is focusing; as discussed below, eye verge may be utilized toestimate depth of focus). The GPS (37), gyros, compass, andaccelerometers (39) may be utilized to provide course and/or fast poseestimates. The camera (16) images and pose, in conjunction with datafrom an associated cloud computing resource, may be utilized to map thelocal world and share user views with a virtual or augmented realitycommunity. While much of the hardware in the display system (14)featured in FIG. 14 is depicted directly coupled to the housing (84)which is adjacent the display (82) and eyes (20) of the user, thehardware components depicted may be mounted to or housed within othercomponents, such as a belt-mounted component, as shown, for example, inFIG. 3. In one embodiment, all of the components of the system (14)featured in FIG. 14 are directly coupled to the display housing (84)except for the image pose processor (36), sensor pose processor (32),and rendering engine (34), and communication between the latter threeand the remaining components of the system (14) may be by wirelesscommunication, such as ultra wideband, or wired communication. Thedepicted housing (84) preferably is head-mounted and wearable by theuser. It may also feature speakers, such as those which may be insertedinto the ears of a user and utilized to provide sound to the user whichmay be pertinent to an augmented or virtual reality experience such asthe jungle sounds referred to in reference to FIG. 13, and microphones,which may be utilized to capture sounds local to the user.

Regarding the projection of light (38) into the eyes (20) of the user,in one embodiment the mini cameras (24) may be utilized to measure wherethe centers of a user's eyes (20) are geometrically verged to, which, ingeneral, coincides with a position of focus, or “depth of focus”, of theeyes (20). A 3-dimensional surface of all points the eyes verge to iscalled the “horopter”. The focal distance may take on a finite number ofdepths, or may be infinitely varying. Light projected from the vergencedistance appears to be focused to the subject eye (20), while light infront of or behind the vergence distance is blurred. Further, it hasbeen discovered that spatially coherent light with a beam diameter ofless than about 0.7 millimeters is correctly resolved by the human eyeregardless of where the eye focuses; given this understanding, to createan illusion of proper focal depth, the eye vergence may be tracked withthe mini cameras (24), and the rendering engine (34) and projectionsubsystem (18) may be utilized to render all objects on or close to thehoropter in focus, and all other objects at varying degrees of defocus(i.e., using intentionally-created blurring). A see-through light guideoptical element configured to project coherent light into the eye may beprovided by suppliers such as Lumus, Inc. Preferably the system (14)renders to the user at a frame rate of about 60 frames per second orgreater. As described above, preferably the mini cameras (24) may beutilized for eye tracking, and software may be configured to pick up notonly vergence geometry but also focus location cues to serve as userinputs. Preferably such system is configured with brightness andcontrast suitable for day or night use. In one embodiment such systempreferably has latency of less than about 20 milliseconds for visualobject alignment, less than about 0.1 degree of angular alignment, andabout 1 arc minute of resolution, which is approximately the limit ofthe human eye. The display system (14) may be integrated with alocalization system, which may involve the GPS element, opticaltracking, compass, accelerometer, and/or other data sources, to assistwith position and pose determination; localization information may beutilized to facilitate accurate rendering in the user's view of thepertinent world (i.e., such information would facilitate the glasses toknow where they are with respect to the real world).

Other suitable display device include but are not limited to desktop andmobile computers, smartphones, smartphones which may be enhancedadditional with software and hardware features to facilitate or simulate3-D perspective viewing (for example, in one embodiment a frame may beremovably coupled to a smartphone, the frame featuring a 200 Hz gyro andaccelerometer sensor subset, two small machine vision cameras with widefield of view lenses, and an ARM processor—to simulate some of thefunctionality of the configuration featured in FIG. 14), tabletcomputers, tablet computers which may be enhanced as described above forsmartphones, tablet computers enhanced with additional processing andsensing hardware, head-mounted systems that use smartphones and/ortablets to display augmented and virtual viewpoints (visualaccommodation via magnifying optics, mirrors, contact lenses, or lightstructuring elements), non-see-through displays of light emittingelements (LCDs, OLEDs, vertical-cavity-surface-emitting lasers, steeredlaser beams, etc), see-through displays that simultaneously allow humansto see the natural world and artificially generated images (for example,light-guide optical elements, transparent and polarized OLEDs shininginto close-focus contact lenses, steered laser beams, etc), contactlenses with light-emitting elements (such as those available fromInnovega, Inc., of Bellevue, Wash., under the tradename loptik®; theymay be combined with specialized complimentary eyeglasses components),implantable devices with light-emitting elements, and implantabledevices that stimulate the optical receptors of the human brain.

With a system such as that depicted in FIGS. 3 and 14, 3-D points may becaptured from the environment, and the pose (i.e., vector and/or originposition information relative to the world) of the cameras that capturethose images or points may be determined, so that these points or imagesmay be “tagged”, or associated, with this pose information. Then pointscaptured by a second camera may be utilized to determine the pose of thesecond camera. In other words, one can orient and/or localize a secondcamera based upon comparisons with tagged images from a first camera.Then this knowledge may be utilized to extract textures, make maps, andcreate a virtual copy of the real world (because then there are twocameras around that are registered). So at the base level, in oneembodiment you have a person-worn system that can be utilized to captureboth 3-D points and the 2-D images that produced the points, and thesepoints and images may be sent out to a cloud storage and processingresource.

They may also be cached locally with embedded pose information (i.e.,cache the tagged images); so the cloud may have on the ready (i.e., inavailable cache) tagged 2-D images (i.e., tagged with a 3-D pose), alongwith 3-D points. If a user is observing something dynamic, he may alsosend additional information up to the cloud pertinent to the motion (forexample, if looking at another person's face, the user can take atexture map of the face and push that up at an optimized frequency eventhough the surrounding world is otherwise basically static).

The cloud system may be configured to save some points as fiducials forpose only, to reduce overall pose tracking calculation. Generally it maybe desirable to have some outline features to be able to track majoritems in a user's environment, such as walls, a table, etc, as the usermoves around the room, and the user may want to be able to “share” theworld and have some other user walk into that room and also see thosepoints. Such useful and key points may be termed “fiducials” becausethey are fairly useful as anchoring points -they are related to featuresthat may be recognized with machine vision, and that can be extractedfrom the world consistently and repeatedly on different pieces of userhardware. Thus these fiducials preferably may be saved to the cloud forfurther use.

In one embodiment it is preferable to have a relatively evendistribution of fiducials throughout the pertinent world, because theyare the kinds of items that cameras can easily use to recognize alocation.

In one embodiment, the pertinent cloud computing configuration may beconfigured to groom the database of 3-D points and any associated metadata periodically to use the best data from various users for bothfiducial refinement and world creation. In other words, the system maybe configured to get the best dataset by using inputs from various userslooking and functioning within the pertinent world. In one embodimentthe database is intrinsically fractal—as users move closer to objects,the cloud passes higher resolution information to such users. As a usermaps an object more closely, that data is sent to the cloud, and thecloud can add new 3-D points and image-based texture maps to thedatabase if they are better than what has been previously stored in thedatabase. All of this may be configured to happen from many userssimultaneously.

As described above, an augmented or virtual reality experience may bebased upon recognizing certain types of objects. For example, it may beimportant to understand that a particular object has a depth in order torecognize and understand such object. Recognizer software objects(“recognizers”) may be deployed on cloud or local resources tospecifically assist with recognition of various objects on either orboth platforms as a user is navigating data in a world. For example, ifa system has data for a world model comprising 3-D point clouds andpose-tagged images, and there is a desk with a bunch of points on it aswell as an image of the desk, there may not be a determination that whatis being observed is, indeed, a desk as humans would know it. In otherwords, some 3-D points in space and an image from someplace off in spacethat shows most of the desk may not be enough to instantly recognizethat a desk is being observed. To assist with this identification, aspecific object recognizer may be created that will go into the raw 3-Dpoint cloud, segment out a set of points, and, for example, extract theplane of the top surface of the desk. Similarly, a recognizer may becreated to segment out a wall from 3-D points, so that a user couldchange wallpaper or remove part of the wall in virtual or augmentedreality and have a portal to another room that is not actually there inthe real world. Such recognizers operate within the data of a worldmodel and may be thought of as software “robots” that crawl a worldmodel and imbue that world model with semantic information, or anontology about what is believed to exist amongst the points in space.Such recognizers or software robots may be configured such that theirentire existence is about going around the pertinent world of data andfinding things that it believes are walls, or chairs, or other items.They may be configured to tag a set of points with the functionalequivalent of, “this set of points belongs to a wall”, and may comprisea combination of point-based algorithm and pose-tagged image analysisfor mutually informing the system regarding what is in the points.

Object recognizers may be created for many purposes of varied utility,depending upon the perspective. For example, in one embodiment, apurveyor of coffee such as Starbucks may invest in creating an accuraterecognizer of Starbucks coffee cups within pertinent worlds of data.Such a recognizer may be configured to crawl worlds of data large andsmall searching for Starbucks coffee cups, so they may be segmented outand identified to a user when operating in the pertinent nearby space(i.e., perhaps to offer the user a coffee in the Starbucks outlet rightaround the corner when the user looks at his Starbucks cup for a certainperiod of time). With the cup segmented out, it may be recognizedquickly when the user moves it on his desk. Such recognizers may beconfigured to run or operate not only on cloud computing resources anddata, but also on local resources and data, or both cloud and local,depending upon computational resources available. In one embodiment,there is a global copy of the world model on the cloud with millions ofusers contributing to that global model, but for smaller worlds orsub-worlds like an office of a particular individual in a particulartown, most of the global world will not care what that office lookslike, so the system may be configured to groom data and move to localcache information that is believed to be most locally pertinent to agiven user. In one embodiment, for example, when a user walks up to adesk, related information (such as the segmentation of a particular cupon his table) may be configured to reside only upon his local computingresources and not on the cloud, because objects that are identified asones that move often, such as cups on tables, need not burden the cloudmodel and transmission burden between the cloud and local resources.Thus the cloud computing resource may be configured to segment 3-Dpoints and images, thus factoring permanent (i.e., generally not moving)objects from movable ones, and this may affect where the associated datais to remain, where it is to be processed, remove processing burden fromthe wearable/local system for certain data that is pertinent to morepermanent objects, allow one-time processing of a location which thenmay be shared with limitless other users, allow multiple sources of datato simultaneously build a databased of fixed and movable objects in aparticular physical location, and segment objects from the background tocreate object-specific fiducials and texture maps.

In one embodiment, the system may be configured to query a user forinput about the identity of certain objects (for example, the system maypresent the user with a question such as, “is that a Starbucks coffeecup?”), so that the user may train the system and allow the system toassociate semantic information with objects in the real world. Anontology may provide guidance regarding what objects segmented from theworld can do, how they behave, etc. In one embodiment the system mayfeature a virtual or actual keypad, such as a wirelessly connectedkeypad, connectivity to a keypad of a smartphone, or the like, tofacilitate certain user input to the system.

The system may be configured to share basic elements (walls, windows,desk geometry, etc) with any user who walks into the room in virtual oraugmented reality, and in one embodiment that person's system will beconfigured to take images from his particular perspective and uploadthose to the cloud. Then the cloud becomes populated with old and newsets of data and can run optimization routines and establish fiducialsthat exist on individual objects.

GPS and other localization information may be utilized as inputs to suchprocessing. Further, other computing systems and data, such as one'sonline calendar or FaceBook account information, may be utilized asinputs (for example, in one embodiment, a cloud and/or local system maybe configured to analyze the content of a user's calendar for airlinetickets, dates, and destinations, so that over time, information may bemoved from the cloud to the user's local systems to be ready for theuser's arrival time in a given destination).

In one embodiment, tags such as QR codes and the like may be insertedinto a world for use with non-statistical pose calculation,security/access control, communication of special information, spatialmessaging, non-statistical object recognition, etc.

In one embodiment, cloud resources may be configured to pass digitalmodels of real and virtual worlds between users, as described above inreference to “passable worlds”, with the models being rendered by theindividual users based upon parameters and textures. This reducesbandwidth relative to the passage of realtime video, allows rendering ofvirtual viewpoints of a scene, and allows millions or more users toparticipate in one virtual gathering without sending each of them datathat they need to see (such as video), because their views are renderedby their local computing resources.

The virtual reality system (“VRS”) may be configured to register theuser location and field of view (together known as the “pose”) throughone or more of the following: realtime metric computer vision using thecameras, simultaneous localization and mapping techniques, maps, anddata from sensors such as gyros, accelerometers, compass, barometer,GPS, radio signal strength triangulation, signal time of flightanalysis, LIDAR ranging, RADAR ranging, odometry, and sonar ranging. Thewearable device system may be configured to simultaneously map andorient. For example, in unknown environments, the VRS may be configuredto collect information about the environment, ascertaining fiducialpoints suitable for user pose calculations, other points for worldmodeling, images for providing texture maps of the world. Fiducialpoints may be used to optically calculate pose. As the world is mappedwith greater detail, more objects may be segmented out and given theirown texture maps, but the world still preferably is representable at lowspatial resolution in simple polygons with low resolution texture maps.Other sensors, such as those discussed above, may be utilized to supportthis modeling effort. The world may be intrinsically fractal in thatmoving or otherwise seeking a better view (through viewpoints,“supervision” modes, zooming, etc.) request high-resolution informationfrom the cloud resources. Moving closer to objects captures higherresolution data, and this may be sent to the cloud, which may calculateand/or insert the new data at interstitial sites in the world model.

Referring to FIG. 16, a wearable system may be configured to captureimage information and extract fiducials and recognized points (52). Thewearable local system may calculate pose using one of the posecalculation techniques mentioned below. The cloud (54) may be configuredto use images and fiducials to segment 3-D objects from more static 3-Dbackground; images provide textures maps for objects and the world(textures may be realtime videos). The cloud resources (56) may beconfigured to store and make available static fiducials and textures forworld registration. The cloud resources may be configured to groom thepoint cloud for optimal point density for registration. The cloudresources (60) may store and make available object fiducials andtextures for object registration and manipulation; the cloud may groompoint clouds for optimal density for registration. The cloud resourcemay be configured (62) to use all valid points and textures to generatefractal solid models of objects; the cloud may groom point cloudinformation for optimal fiducial density. The cloud resource (64) may beconfigured to query users for training on identity of segmented objectsand the world; an ontology database may use the answers to imbue objectsand the world with actionable properties.

The following specific modes of registration and mapping feature theterms “O-pose”, which represents pose determined from the optical orcamera system; “s-pose”, which represents pose determined from thesensors (i.e., such as a combination of GPS, gyro, compass,accelerometer, etc data, as discussed above); and “MLC”, whichrepresents the cloud computing and data management resource.

1. Orient: make a basic map of a new environment

Purpose: establish pose if environment is not mapped or (the equivalent)if not connected to the MLC.

-   -   Extract points from image, track from frame to frame,        triangulate fiducials using S-pose.    -   Uses S-pose because there are no fiducials    -   Filter out bad fiducials based on persistence.    -   This is the most basic mode: it will always work for        low-precision pose. With a little time and some relative motion        it will establish a minimum fiducial set for O-pose and/or        mapping.    -   Jump out of this mode as soon as O-pose is reliable.

2. Map and O-Pose: map an environment

Purpose: establish high-precision pose, map the environment, and providethe map (with images) to the MLC.

-   -   Calculate O-pose from mature world fiducials. Use S-pose as        check of O-pose solution and to speed computation (O-pose is a        non-linear gradient search).    -   Mature fiducials may come from MLC, or be those locally        determined.    -   Extract points from image, track from frame to frame,        triangulate fiducials using O-pose.    -   Filter out bad fiducials based on persistence.    -   Provide MLC with fiducials and pose-tagged images.    -   Last three steps need not happen real-time.

3. O-Pose: determine pose

Purpose: establish high-precision pose in an already mapped environmentusing minimum processing power.

-   -   Use historic S- and O-pose (n−1, n−2, n−3, etc.) to estimate        pose at n.    -   Use pose at n to project fiducials into image captured at n,        then create image mask from the projection.    -   Extract points from the masked regions (processing burden        greatly reduced by only searching/extracting points from the        masked subsets of image).    -   Calculate O-pose from extracted points and mature world        fiducials.    -   Use S- and O-pose at n to estimate pose at n+1.    -   Option: provide pose-tagged images/video to MLC cloud.

4. Super Res: determine super resolution imagery and fiducials

Purpose: create super-resolution imagery and fiducials.

-   -   Composite pose-tagged images to create super-resolution images.    -   Use super-resolution images to enhance fiducial position        estimation.    -   Iterate O-pose estimates from super resolution fiducials and        imagery.    -   Option: Loop the above steps on a wearable device (in real time)        or the MLC (for better world).

In one embodiment, the VLS system may be configured to have certain basefunctionality, as well as functionality facilitated by “apps” orapplications that may be distributed through the VLS to provide certainspecialized functionalities. For example, the following apps may beinstalled to the subject VLS to provide specialized functionality:

Painterly renderings app. Artists create image transforms that representthe world they see it. Users enable these transforms, thus viewing theworld “through” the artist's eyes.

Table top modeling app. Users “build” objects from physical objects puton a table.

Virtual presence app. Users pass virtual model of space to other user,who then moves around space using virtual avatar.

Avatar emotion app. Measurements of subtle voice inflection, minor headmovement, body temperature, heart rate, etc. animate subtle effects onvirtual-presence avatars. Digitizing human state information and passingthat to remote avatar uses less bandwidth then video. Additionally, suchdata is map-able to non-human avatars capable of emotion. Ex. A dogavatar can show excitement by wagging its tail based on excited vocalinflections.

An efficient mesh type network may be desirable for moving data, asopposed to sending everything back to a server. Many mesh networks,however, have suboptimal performance because positional information andtopology is not well characterized. In one embodiment, the system may beutilized to determine the location of all users with relatively highprecision, and thus a mesh network configuration may be utilized forhigh performance.

In one embodiment the system may be utilized for searching. Withaugmented reality, for example, users will generate and leave contentrelated to many aspects of the physical world. Much of this content isnot text, and thus is not easily searched by typical methods. The systemmay be configured to provide a facility for keeping track of personaland social network content for searching and reference purposes.

In one embodiment, if the display device tracks 2-D points throughsuccessive frames, then fits a vector-valued function to the timeevolution of those points, it is possible to sample the vector valuedfunction at any point in time (e.g. between frames) or at some point inthe near future (by projecting the vector-valued function forward intime. This allows creation of high-resolution post-processing, andprediction of future pose before the next image is actual captured(e.g., doubling the registration speed is possible without doubling thecamera frame rate).

For body-fixed rendering (as opposed to head-fixed or world-fixedrenderings) an accurate view of body is desired. Rather than measuringthe body, in one embodiment is possible to derive its location throughthe average position of a users head. If the user's face points forwardmost of the time, a multi-day average of head position will reveal thatdirection. In conjunction with the gravity vector, this provides areasonably stable coordinate frame for body-fixed rendering. Usingcurrent measures of head position with respect to this long-durationcoordinate frame allows consistent rendering of objects on/around ausers body—with no extra instrumentation. For implementation of thisembodiment, single register averages of head direction-vector may bestarted, and a running sum of data divided by delta-t will give currentaverage head position. Keeping five or so registers, started on day n−5,day n−4, day n−3, day n−2, day n−1 allows use of rolling averages ofonly the past “n” days.

In one embodiment, a scene may be scaled down and presented to a user ina smaller-than-actual space. For example, in a situation wherein thereis a scene that must be rendered in a huge space (i.e., such as a soccerstadium), there may be no equivalent huge space present, or such a largespace may be inconvenient to a user. In one embodiment the system may beconfigured to reduce the scale of the scene, so that the user may watchit in miniature. For example, one could have a gods-eye-view video game,or a world championship soccer game, play out in an unscaled field—orscaled down and presented on a living room floor. The system may beconfigured to simply shift the rendering perspective, scale, andassociated accommodation distance.

The system may also be configured to draw a user's attention to specificitems within a presented scene by manipulating focus of virtual oraugmented reality objects, by highlighting them, changing the contrast,brightness, scale, etc.

Preferably the system may be configured to accomplish the followingmodes:

Open space rendering:

Grab key points from structured environment, then fill in the spacebetween with ML renderings.

Potential venues: stages, output spaces, large indoor spaces (stadiums).

Object wrapping:

Recognize 3D object in the real world, then augment them

“Recognition” here means identifying a 3D blob with high enoughprecision to anchor imagery to.

There are two types of recognition: 1) Classifying the type of an object(ex. a “face”); 2) Classifying a particular instance of an object (ex.Joe, a person).

Build recognizers software objects for various things: walls, ceilings,floors, faces, roads, sky, skyscrapers, ranch houses, tables, chairs,cars, road signs, billboards, doors, windows, bookshelves, etc

Some recognizers are Type I, and have generic functionality, e.g. “putmy video on that wall”, “that is a dog”

Other recognizers are Type II, and have specific functionality, e.g. “myTV is on my living room wall 3.2 feet from the ceiling”, “that is Fido”(this is a more capable version of the generic recognizer)

Building recognizer as software objects allows metered release offunctionality, and finer grained control of experience

Body centered rendering

Render virtual objects fixed to the users body.

Some things should float around the user's body, like a digitaltoolbelt.

This requires knowing where the body is, rather than just the head. Mayget body position reasonably accurate by having a long-term average ofusers head position (heads usually point forward parallel to theground).

A trivial case is objects floating around the head. Transparency/cutaway

For Type II recognized objects, show cut-aways

Link Type II recognized objects to an online database of 3D models.

Should start with objects that have commonly available 3D models, suchas cars and public utilities. Virtual presence

Paint remote people's avatars into open spaces.

-   -   A subset of “open space rendering” (above).    -   Users create rough geometry of local environment and iteratively        send both geometry and texture maps to others.    -   Users must grant permission for others to enter their        environment.    -   Subtle voice queues, hand tracking, and head motion are sent to        remote avatar. Avatar is animated from these fuzzy inputs.    -   The above minimize bandwidth.

Make a wall a “portal” to another room

-   -   As with other method, pass geometry and texture map.    -   Instead of showing avatar in local room, designate recognized        object (e.g. a wall) as a portal to the other's environment. In        this way multiple people could sit in their own rooms, looking        “through” walls into the environments of others.        Virtual viewpoints

Dense digital model of area is created when a group of cameras (people)view a scene from different perspectives. This rich digital model isrenderable from any vantage point that at least one camera can see.

Example. People at a wedding. Scene is jointly modeled by all attendees.

Recognizers differentiate and texture map stationary objects differentlythan moving ones (e.g. walls have stable texture map, people have higherfrequency moving texture maps.)

With rich digital model updated in real time, scene is renderable fromany perspective. Attendee in back can fly in the air to the front rowfor a better view.

Attendees can show their moving avatar, or have their perspectivehidden.

Off-site attendees can find a “seat” either with their avatar or if theorganizers permit, invisibly.

Likely requires extremely high bandwidth. Notionally, high frequencydata is steamed through the crowd on a high-speed local wireless. Lowfrequency data comes from the MLC.

Because all attendees have high precision position information, makingan optimal routing path for local networking is trivial.

Messaging

Simple silent messaging may be desirable

For this and other applications, it may be desirable to have a fingerchording keyboard.

Tactile glove solutions may offer enhanced performance. Full VirtualReality (VR):

With vision system darkened, show a view not overlaying on the realworld.

Registration system is still necessary to track head position.

“Couch mode” allows user to fly.

“Walking mode” re-renders objects in the real world as virtual ones souser does not collide with real world.

Rendering body parts is essential for suspension of disbelieve. Thisimplies having method for tracking and rendering body parts in FOV.

Non-see through visor is a form of VR with many image enhancementadvantages not possible with direct overlay

Wide FOV, perhaps even the ability to look to rear

Various forms of “super vision”: telescope, see through, infrared, God'seye, etc.

In one embodiment a system for virtual and/or augmented user experienceis configured such that remote avatars associated with users may beanimated based at least in part upon data on a wearable device withinput from sources such as voice inflection analysis and facialrecognition analysis, as conducted by pertinent software modules. Forexample, referring back to FIG. 12, the bee avatar (2) may be animatedto have a friendly smile based upon facial recognition of a smile uponthe user's face, or based upon a friendly tone of voice or speaking, asdetermined by software configured to analyze voice inputs to microphoneswhich may capture voice samples locally from the user. Further, theavatar character may be animated in a manner in which the avatar islikely to express a certain emotion. For example, in an embodimentwherein the avatar is a dog, a happy smile or tone detected by systemlocal to the human user may be expressed in the avatar as a wagging tailof the dog avatar.

Various exemplary embodiments of the invention are described herein.Reference is made to these examples in a non-limiting sense. They areprovided to illustrate more broadly applicable aspects of the invention.Various changes may be made to the invention described and equivalentsmay be substituted without departing from the true spirit and scope ofthe invention. In addition, many modifications may be made to adapt aparticular situation, material, composition of matter, process, processact(s) or step(s) to the objective(s), spirit or scope of the presentinvention. Further, as will be appreciated by those with skill in theart that each of the individual variations described and illustratedherein has discrete components and features which may be readilyseparated from or combined with the features of any of the other severalembodiments without departing from the scope or spirit of the presentinventions. All such modifications are intended to be within the scopeof claims associated with this disclosure.

The invention includes methods that may be performed using the subjectdevices. The methods may comprise the act of providing such a suitabledevice. Such provision may be performed by the end user. In other words,the “providing” act merely requires the end user obtain, access,approach, position, set-up, activate, power-up or otherwise act toprovide the requisite device in the subject method. Methods recitedherein may be carried out in any order of the recited events which islogically possible, as well as in the recited order of events.

Exemplary aspects of the invention, together with details regardingmaterial selection and manufacture have been set forth above. As forother details of the present invention, these may be appreciated inconnection with the above-referenced patents and publications as well asgenerally known or appreciated by those with skill in the art. The samemay hold true with respect to method-based aspects of the invention interms of additional acts as commonly or logically employed.

In addition, though the invention has been described in reference toseveral examples optionally incorporating various features, theinvention is not to be limited to that which is described or indicatedas contemplated with respect to each variation of the invention. Variouschanges may be made to the invention described and equivalents (whetherrecited herein or not included for the sake of some brevity) may besubstituted without departing from the true spirit and scope of theinvention. In addition, where a range of values is provided, it isunderstood that every intervening value, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range, is encompassed within the invention.

Also, it is contemplated that any optional feature of the inventivevariations described may be set forth and claimed independently, or incombination with any one or more of the features described herein.Reference to a singular item, includes the possibility that there areplural of the same items present. More specifically, as used herein andin claims associated hereto, the singular forms “a,” “an,” “said,” and“the” include plural referents unless the specifically stated otherwise.In other words, use of the articles allow for at least one of thesubject item in the description above as well as claims associated withthis disclosure. It is further noted that such claims may be drafted toexclude any optional element. As such, this statement is intended toserve as antecedent basis for use of such exclusive terminology as“solely,” “only” and the like in connection with the recitation of claimelements, or use of a “negative” limitation.

Without the use of such exclusive terminology, the term “comprising” inclaims associated with this disclosure shall allow for the inclusion ofany additional element—irrespective of whether a given number ofelements are enumerated in such claims, or the addition of a featurecould be regarded as transforming the nature of an element set forth insuch claims. Except as specifically defined herein, all technical andscientific terms used herein are to be given as broad a commonlyunderstood meaning as possible while maintaining claim validity.

The breadth of the present invention is not to be limited to theexamples provided and/or the subject specification, but rather only bythe scope of claim language associated with this disclosure.

1. (canceled)
 2. A method comprising: receiving, at a computer networkcomprising one or more computing devices, via a user-sensing system,from a first user, a first input; receiving, via an environment-sensingsystem, from a local environment of a first user device, a second input;generating second virtual world data based on first virtual world dataand further based on at least one of the first input and the secondinput; and communicating the second virtual world data to the first userdevice, wherein: the first user device is configured to present, basedon the second virtual world data, virtual content to the first user,presenting the virtual content to the first user comprises presenting avisual rendering of the virtual content in a 3D format, and presentingthe visual rendering of the virtual content in the 3D format comprisespresenting the visual rendering on a display based on a position andorientation of the first user device.
 3. The method of claim 2, wherein:generating the second virtual world data comprises generating a virtualobject based on at least one of the first input and the second input,and the virtual object is associated with a physical object in the localenvironment.
 4. The method of claim 2, wherein the computer networkcomprises a mobile phone.
 5. The method of claim 4, wherein the firstuser device is configured to removably couple with the mobile phone. 6.The method of claim 2, wherein the computer network comprises a personalcomputer.
 7. The method of claim 2, wherein presenting the visualrendering on the display comprises presenting the visual rendering at aposition on the display determined based on a location of a firstphysical object.
 8. The method of claim 2, wherein presenting the visualrendering on the display comprises presenting the visual rendering at aposition on the display determined based on the position and orientationof the first user device.
 9. The method of claim 2, wherein the positionof the first user device comprises a geospatial location of the firstuser device.
 10. The method of claim 2, wherein presenting the virtualcontent to the first user further comprises presenting the virtualcontent to the first user in response to a determination that a locationof the first user device corresponds to a first location.
 11. The methodof claim 2, wherein presenting the virtual content to the first userfurther comprises presenting the virtual content to the first user inresponse to detecting a first physical object.
 12. The method of claim2, wherein the first user device comprises a wearable device comprisingthe display, and wherein the display comprises a head-mountedsee-through display.
 13. The method of claim 12, wherein the wearabledevice is configured to communicate with the computer network via amobile phone.
 14. A non-transitory computer-readable medium containinginstructions which, when executed by one or more processors, cause theone or more processors to perform a method comprising: receiving, at afirst user device, from a computer network comprising one or morecomputing devices, second virtual world data, wherein: the secondvirtual world data is generated by one or more of the one or morecomputing devices based on first virtual world data and further based onat least one of: a first input received from a first user via auser-sensing system, and a second input received from a localenvironment of the first user device; and presenting, based on thesecond virtual world data, virtual content to the first user, wherein:presenting the virtual content to the first user comprises presenting avisual rendering of the virtual content in a 3D format, and presentingthe visual rendering of the virtual content in the 3D format comprisespresenting the visual rendering on a display based on a position andorientation of the first user device.
 15. The non-transitorycomputer-readable medium of claim 14, wherein: generating the secondvirtual world data comprises generating a virtual object based on atleast one of the first input and the second input, and the virtualobject is associated with a physical object in the local environment.16. The non-transitory computer-readable medium of claim 14, wherein thecomputer network comprises a mobile phone.
 17. The non-transitorycomputer-readable medium of claim 16, wherein the first user device isconfigured to removably couple with the mobile phone.
 18. Thenon-transitory computer-readable medium of claim 14, wherein thecomputer network comprises a personal computer.
 19. The non-transitorycomputer-readable medium of claim 14, wherein presenting the visualrendering on the display comprises presenting the visual rendering at aposition on the display determined based on a location of a firstphysical object.
 20. The non-transitory computer-readable medium ofclaim 14, wherein presenting the visual rendering on the displaycomprises presenting the visual rendering at a position on the displaydetermined based on the position and orientation of the first userdevice.
 21. The non-transitory computer-readable medium of claim 14,wherein the position of the first user device comprises a geospatiallocation of the first user device.
 22. The non-transitorycomputer-readable medium of claim 14, wherein presenting the virtualcontent to the first user further comprises presenting the virtualcontent to the first user in response to a determination that a locationof the first user device corresponds to a first location.
 23. Thenon-transitory computer-readable medium of claim 14, wherein presentingthe virtual content to the first user further comprises presenting thevirtual content to the first user in response to detecting a firstphysical object.
 24. The non-transitory computer-readable medium ofclaim 14, wherein the first user device comprises a wearable devicecomprising the display, and wherein the display comprises a head-mountedsee-through display.
 25. The non-transitory computer-readable medium ofclaim 24, wherein the wearable device is configured to communicate withthe computer network via a mobile phone.